Image classification using CNNs in Keras¶Context:¶Can you differentiate a weed from a crop seedling?
The ability to do so effectively can mean better crop yields and better stewardship of the environment.
The Aarhus University Signal Processing group, in collaboration with University of Southern Denmark, has recently released a dataset containing images of unique plants belonging to 12 species at several growth stages.
Learning Outcomes:¶Guide to solve the project seamlessly:¶Here are the points which will help you to solve the problem efficiently:
Data Description:¶You are provided with a dataset of images of plant seedlings at various stages of grown. Each image has a filename that is its unique id. The dataset comprises 12 plant species. The goal of the project is to create a classifier capable of determining a plant's species from a photo.
Dataset:¶The data file names are:
The original files are from Kaggle. Due to the large volume of data, the images were converted to images.npy file and the labels are also put into the Labels.csv. So that you can work on the data/project seamlessly without worrying about the high data volume. The following code was used to convert the large dataset of images to numpy array:
The data before conversion was downloaded from Kaggle project site: https://www.kaggle.com/c/plant-seedlings-classification/data?select=train
#This code is an illustration of how the data for this project was generated.
# Import necessary libraries.
#import math
#import numpy as np
#import pandas as pd
#from glob import glob
#data_path = '/content/drive/My Drive/Colab Notebooks/data/plant_seedlings/train.zip'
#!mkdir dataset
# Extract the files from dataset to temp_train and temp_test folders (as the dataset is a zip file.)
#from zipfile import ZipFile
#with ZipFile(data_path, 'r') as zip:
# zip.extractall('./dataset')
#path = "/content/dataset/*/*.*" # The path to all images in training set. (* means include all folders and files.)
#files = glob(path)
#trainImg = [] # Initialize empty list to store the image data as numbers.
#trainLabel = [] # Initialize empty list to store the labels of images
#j = 1
#num = len(files)
# Obtain images and resizing, obtain labels
#for img in files:
# Append the image data to trainImg list.
# Append the labels to trainLabel list.
# print(str(j) + "/" + str(num), end="\r")
# trainImg.append(cv2.resize(cv2.imread(img), (128, 128))) # Get image (with resizing to 128x128)
# trainLabel.append(img.split('/')[-2]) # Get image label (folder name contains the class to which the image belong)
# j += 1
#trainImg = np.asarray(trainImg) # Train images set
#trainLabel = pd.DataFrame(trainLabel, columns=["Label"]) # Train labels set
#print(trainImg.shape)
#print(trainLabel.shape)
#trainLabel.to_csv('Labels.csv', index=False)
#np.save('plantimages', trainImg)
Steps and tasks:¶Deliverable – 1:¶Import the libraries, load dataset, print shape of data, visualize the images in dataset. (5 Marks)
Deliverable – 2:¶Data Pre-processing: (15 Marks)
Deliverable – 3:¶Make data compatible: (10 Marks)
Deliverable – 4:¶Building CNN (15 Marks)
Deliverable – 5:¶Fit and evaluate model and print confusion matrix. (10 Marks)
Deliverable – 6:¶Visualize predictions for x_test[2], x_test[3], x_test[33], x_test[36], x_test[59]. (5 Marks)
Note:¶Do not download the dataset from Kaggle, as:
Deliverable – 1: ¶from google.colab import drive
#drive.mount('/content/drive')
%tensorflow_version 2.x
import tensorflow
tensorflow.__version__
#import libraries
import numpy as np
import pandas as pd
import warnings
warnings.filterwarnings("ignore")
import matplotlib.pyplot as plt
import seaborn as sns
import matplotlib.cm as cm
%matplotlib inline
from sklearn import metrics
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Conv2D, Dropout, MaxPooling2D, Flatten
from keras import models, layers, callbacks
from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import confusion_matrix, classification_report
import cv2
from google.colab.patches import cv2_imshow
from sklearn.preprocessing import LabelEncoder
from tensorflow.keras.utils import to_categorical
#Read Data
from google.colab import drive
drive.mount('/gdrive')
%cd /gdrive/'My Drive'/'Colab Notebooks'
labels = pd.read_csv('Labels.csv')
labels.head(5)
images = np.load('images.npy')
#Stastical Analysis
print(images.shape)
print(labels.shape)
# Following code gives the first image. The code is commented for better visibility of notebook.
#images[0]
# Following gives the first row in the first image. The code is commented for better visibility of notebook.
#images[0][0]
#Following gives the first pixel in the first image.
images[0][0][0]
#Step4: Find number of unique values in each column. Analyse outcome for categorical variables.
labels.nunique() # Number of unique values in a column
# value counts gives us how many times does the value
print(labels['Label'].value_counts(normalize=True))
print(labels['Label'].value_counts())
sns.countplot(y=labels["Label"])
Observation:¶from random import randint
from keras.preprocessing import image
# Find the start and end indexes of each classification
fig= plt.figure(figsize= (18, 40))
fig.suptitle('Random Samples From Each Class', fontsize=14, y=.92, horizontalalignment='center', weight='bold')
columns = 5
rows = 13
labelss = list(labels.Label.unique())
for i in range(12):
lab = labelss[i]
a = list(labels[labels['Label']==lab].index.values)
fst_index = a[0]
lst_index = a[-1]
for j in range(1,6):
fig.add_subplot(rows, columns, i*5+j)
plt.axis('off')
if j==1:
plt.text(0.0, 0.5,lab, fontsize=15, wrap=True)
continue
r = randint(fst_index, lst_index)
random_image= images[r]
plt.imshow(random_image)
plt.show()
Observation:¶Deliverable – 2:¶Approach:¶Data Pre-processing is performed as follows:
# 1. Normalization:
# First convert the datatype of each RGB value in each pixel to float. Currently its a whole number
# ranging between 0 - 255
images = np.load('images.npy')
images1 = images.astype('float32')
# For normalization, divide each value by 255
images1 /= 255
images1[3731][0][0]
# Visualize images after normalization
fig= plt.figure(figsize= (18, 40))
fig.suptitle('Random Samples From Each Class', fontsize=14, y=.92, horizontalalignment='center', weight='bold')
columns = 5
rows = 13
labelss = list(labels.Label.unique())
for i in range(12):
lab = labelss[i]
a = list(labels[labels['Label']==lab].index.values)
fst_index = a[0]
lst_index = a[-1]
for j in range(1,6):
fig.add_subplot(rows, columns, i*5+j)
plt.axis('off')
if j==1:
plt.text(0.0, 0.5,lab, fontsize=15, wrap=True)
continue
r = randint(fst_index, lst_index)
random_image= images1[r]
plt.imshow(random_image)
plt.show()
#Check how the images blur when different filters are applied
blur_img = []
for i in images1:
blur_img.append(cv2.GaussianBlur(i, (15, 15), 0))
# Visualize Images After Guassian Blurring
fig= plt.figure(figsize= (18, 40))
fig.suptitle('Random Samples From Each Class', fontsize=14, y=.92, horizontalalignment='center', weight='bold')
columns = 5
rows = 13
labelss = list(labels.Label.unique())
for i in range(12):
lab = labelss[i]
a = list(labels[labels['Label']==lab].index.values)
fst_index = a[0]
lst_index = a[-1]
for j in range(1,6):
fig.add_subplot(rows, columns, i*5+j)
plt.axis('off')
if j==1:
plt.text(0.0, 0.5,lab, fontsize=15, wrap=True)
continue
r = randint(fst_index, lst_index)
random_image= blur_img[r]
plt.imshow(random_image)
plt.show()
blur_img = []
for i in images1:
blur_img.append(cv2.GaussianBlur(i, (5, 5), 0))
plt.imshow(blur_img[3731])
# Visualize Images After Guassian Blurring
fig= plt.figure(figsize= (18, 40))
fig.suptitle('Random Samples From Each Class', fontsize=14, y=.92, horizontalalignment='center', weight='bold')
columns = 5
rows = 13
labelss = list(labels.Label.unique())
for i in range(12):
lab = labelss[i]
a = list(labels[labels['Label']==lab].index.values)
fst_index = a[0]
lst_index = a[-1]
for j in range(1,6):
fig.add_subplot(rows, columns, i*5+j)
plt.axis('off')
if j==1:
plt.text(0.0, 0.5,lab, fontsize=15, wrap=True)
continue
r = randint(fst_index, lst_index)
random_image= blur_img[r]
plt.imshow(random_image)
plt.show()
blur_img = []
for i in images1:
blur_img.append(cv2.GaussianBlur(i, (3, 3), 0))
plt.imshow(blur_img[3731])
# Visualize Images After Guassian Blurring
fig= plt.figure(figsize= (18, 40))
fig.suptitle('Random Samples From Each Class', fontsize=14, y=.92, horizontalalignment='center', weight='bold')
columns = 5
rows = 13
labelss = list(labels.Label.unique())
for i in range(12):
lab = labelss[i]
a = list(labels[labels['Label']==lab].index.values)
fst_index = a[0]
lst_index = a[-1]
for j in range(1,6):
fig.add_subplot(rows, columns, i*5+j)
plt.axis('off')
if j==1:
plt.text(0.0, 0.5,lab, fontsize=15, wrap=True)
continue
r = randint(fst_index, lst_index)
random_image= blur_img[r]
plt.imshow(random_image)
plt.show()
Observations:¶For observations, area between Y = 60 to 80 and X = 100 to 120 (area referring to tip of leaf is considered)
It is worth noting that there is some noise at the right hand side of the image.
Following Cell is for Illustration Purpose Only and Doesn't Impact Model / Data.¶# Illustration purpose only:
# This is an attempt to reduce noise from the images to highlight leaves.
# However, even after using this approach, model performance didn't improve.
# Therefore, it is now given for illustration purpose only. This can be improved in future iterations.
# Find the start and end indexes of each classification
lower_bound= (24, 50, 0)
upper_bound= (55, 255, 255)
fig= plt.figure(figsize=(12, 16))
fig.suptitle('Random Pre-Processed Image From Each Class', fontsize=14, y=.92, horizontalalignment='center', weight='bold')
columns = 5
rows = 13
labelss = list(labels.Label.unique())
for i in range(12):
lab = labelss[i]
a = list(labels[labels['Label']==lab].index.values)
fst_index = a[0]
lst_index = a[-1]
for j in range(1,3):
r = randint(fst_index, lst_index)
random_image= images[r]
img= random_image
img= cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
#img= cv2.resize(img, (150, 150))
hsv_img= cv2.cvtColor(img, cv2.COLOR_RGB2HSV)
mask = cv2.inRange(hsv_img, lower_bound, upper_bound)
result = cv2.bitwise_and(img, img, mask=mask)
#plt.imshow(result)
fig.add_subplot(6, 4, i*2+1)
plt.imshow(img)
plt.axis('off')
fig.add_subplot(6, 4, i*2+2)
plt.imshow(result)
plt.axis('off')
plt.show()
Following Cell is for Illustration Purpose Only and Doesn't Impact Model / Data.¶# Illustration purpose only:
# Code mentioned in above cell works well to separate the leaf part of image from it's background.
# Therefore, this is an attempt to reduce noise from the images to highlight leaves.
# However, even after using this approach, model performance didn't improve.
# Therefore, it is now given for illustration purpose only. This can be improved in future iterations.
images = np.load('images.npy')
# First convert the datatype of each RGB value in each pixel to float. Currently its a whole number
# ranging between 0 - 255
images1 = images.astype('float32')
# For normalization, divide each value by 255
images1 /= 255
blur_img1 = []
for i in images1:
hsv_img= cv2.cvtColor(i, cv2.COLOR_RGB2HSV)
mask = cv2.inRange(hsv_img, (24, 50, 0), (55, 255, 255))
result = cv2.bitwise_and(i, i, mask=mask)
result= result.astype('float64')
#blur_img1.append(result)
blur_img1.append(cv2.GaussianBlur(result, (3, 3), 0))
Deliverable – 3: ¶Approach:¶Make data compatible: (10 Marks)
#Step1: Get all the labels
y = labels.Label
labels.nunique()
#Print first label in the array
print(labels.iloc[0])
y.iloc[0]
Important Note:¶Once labels in "Y-axis" are enumerated, the word names in classes will be replaced by numbers. It will make it difficult to understand and debug the model. Therefore, following code creates a dataframe containing the mapping between word and numeric values.
#Step2: Encode labels. Note: Step 3 is after a few cells.
label_encoder = LabelEncoder()
# First get all the labels in a variable called y for understanding purposes.
y = labels.Label
#print(y[1])
# Convert y from categorical (word) labels to numeric labels. This creates a numpy array
y1 = label_encoder.fit_transform(y)
#print(y1[1])
#print(type(y1))
# Convert above numpy array to Series
y_numeric = pd.Series(y1)
# Convert both series into a dictionary
df_y_dict = {'Label': y, 'Class_number': y_numeric}
print(type(df_y_dict))
# Finally convert dictionary into a dataframe
df_y = pd.DataFrame(df_y_dict)
# Drop duplicates from dataframe to retain 1:1 mapping between labels and their class number
df_y.drop_duplicates(inplace=True)
# Change y to reflect the numeric values. Optional step.
# It has been carried out to maintain reference integrity of variable y
y = y1
df_y
Following Cell is for Illustration Purpose Only and Doesn't Impact Model / Data.¶Approach:¶As observed earlier, some classes are over-represented in data while some are under represented. Therefore, following code is an attempt to provide custom class weights to data so that low frequency classes can get higher weightage. Following is the approach:
# Calculate Class Weights:
y_num = y_numeric.value_counts(normalize=True)
#print(y_num)
# This dictionary stores class and its weight. The same can be passed on while training model.
# Model paramter accepts dictionary input only for class_weight
class_weight = {}
for i in range(12):
class_weight[i] = (1/y_num[i])
sum = 0
for val in class_weight.values():
sum += val
norm_factor= sum / len(class_weight)
#print(norm_factor)
for i in range(12):
class_weight[i] = (1/y_num[i])/norm_factor
print(class_weight)
#Split train and test set. Validation split is not performed here.
#Validation split will be taken care while training the model.
X_train, X_test, y_train, y_test = train_test_split(blur_img, y, test_size=0.30, random_state=42)
X_train = np.array(X_train)
X_test = np.array(X_test)
y_train = np.array(y_train)
y_test = np.array(y_test)
#Check the shape of train and test sets
print(X_train.shape)
print(y_train.shape)
print(X_test.shape)
print(y_test.shape)
#Note that conversion of labels to one-hot-vectors is not yet completed.
#Print y_train[0]
print(y_train[0])
df_y[df_y['Class_number'] == y_train[0]]
Obserations:¶The data is not yet ready for using in model. Need to work on Y-train and Y-test data further.
First re-split Y Test to form two new datasets: validation and Testing.
Use to_categorical function on Y train and Y validation data to convert them to categorical.
print("X_train shape:", X_train.shape)
print("Images in X_train:", X_train.shape[0])
print("Images in X_test:", X_test.shape[0])
print("Max value in X_train:", X_train.max())
print("Min value in X_train:", X_train.min())
#Step3: Convert labels to one-hot-vectors.
# Use to_categorical function on Y train and Y test data to convert them to categorical.
#from tensorflow.keras.utils import to_categorical
y_train = to_categorical(y_train, num_classes=12)
#y_test = to_categorical(y_test, num_classes=12)
print("Shape of y_train:", y_train.shape)
print("One value of y_train:", y_train[0])
#Print data shape again to confirm the conversion of Y data
print(X_train.shape)
print(y_train.shape)
print(X_test.shape)
print(y_test.shape)
#Split X_test further into testing and validation sets.
X_test_t, X_validation, y_test_t, y_validation = train_test_split(X_test, y_test, test_size=0.50, random_state=42)
y_validation = to_categorical(y_validation, num_classes=12)
#Note: Y test need not undergo to_categorical transformation.
print(X_test_t.shape)
print(y_test_t.shape)
print(X_validation.shape)
print(y_validation.shape)
Observations:¶70% training data and 30% testing data.
Testing data is further split into 50% test and 50% validation sets.
Y-Data (train and validation) was converted to categorical, and now we are ready to train and test the model.
Similar number of records in training testing and validation sets.
Deliverable – 4: ¶Approach:¶Building CNN (15 Marks)
Note: I have tried three callbacks, "EarlyStopping", "ReduceLROnPlateau", and "ModelCheckpoint" during the model building process. However, "ModelCheckpoint" performed better, and therefore is used for final model selection process. Other callback methods are given here for illustration purposes only.
# Initialize the model
model = models.Sequential()
# Add a Convolutional Layer with 32 filters of size 3X3 and activation function as 'relu'
model.add(Conv2D(filters=32, kernel_size=(3,3), padding='same', activation="relu", input_shape=(128, 128, 3)))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.1))
# Add a Convolutional Layer with 32 filters of size 3X3 and activation function as 'relu'
model.add(Conv2D(filters=32, kernel_size=(3,3), padding='same', activation="relu", input_shape=(128, 128, 3)))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.2))
# Add a Convolutional Layer with 64 filters of size 3X3 and activation function as 'relu'
model.add(Conv2D(filters=64, kernel_size=(3,3), padding='same', activation="relu", input_shape=(128, 128, 3)))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.2))
# Add a Convolutional Layer with 64 filters of size 3X3 and activation function as 'relu'
model.add(Conv2D(filters=64, kernel_size=(3,3), padding='same', activation="relu", input_shape=(128, 128, 3)))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.2))
# Add a Convolutional Layer with 64 filters of size 3X3 and activation function as 'relu'
model.add(Conv2D(filters=64, kernel_size=(3,3), padding='same', activation="relu", input_shape=(128, 128, 3)))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.2))
# Add a Convolutional Layer with 128 filters of size 3X3 and activation function as 'relu'
model.add(Conv2D(filters=128, kernel_size=3, padding='same', activation="relu", input_shape=(128, 128, 3)))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.3))
# Add a Convolutional Layer with 128 filters of size 3X3 and activation function as 'relu'
model.add(Conv2D(filters=128, kernel_size=3, padding='same', activation="relu", input_shape=(128, 128, 3)))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Dropout(0.1))
# Flatten the layer
model.add(Flatten())
model.add(Dropout(0.1))
# Add Fully Connected Layer with 512 units and activation function as 'relu'
model.add(Dense(512, activation="relu"))
model.add(Dropout(0.4))
# Add Fully Connected Layer with 256 units and activation function as 'relu'
model.add(Dense(256, activation="relu"))
model.add(Dropout(0.4))
#model.add(Dense(32, activation="relu"))
#model.add(Dropout(0.1))
#Add Fully Connected Layer with 10 units and activation function as 'softmax'
model.add(Dense(12, activation="softmax"))
#Print model summary
model.summary()
# Set Optimizer to Adam And loss to "categorical_crossentropy"
opt= keras.optimizers.Adam(lr=0.0005, amsgrad=True)
# Compile the model
model.compile(loss="categorical_crossentropy", metrics=["accuracy"], optimizer=opt)
# Use one of the following callbacks.
# EarlyStopping: Didn't work the best, unable to set an optimal value for patience. Sometimes it goes low and sometimes very high
callback = tensorflow.keras.callbacks.EarlyStopping(monitor='val_accuracy', patience=2, min_delta=0.01)
# ReduceLROnPlateau: Didn't work the best.
reduce_lr = tensorflow.keras.callbacks.ReduceLROnPlateau(monitor='val_loss', factor=0.2,
patience=5, min_lr=0.001)
# This worked the best since number of epochs is not large.
best_cb= tensorflow.keras.callbacks.ModelCheckpoint('model_best.h5',
monitor='val_loss',
verbose=1,
save_best_only=True,
save_weights_only=False,
mode='auto')
# Fit the model
history = model.fit(x=X_train
, y=y_train, steps_per_epoch= 190
, epochs=75
#, class_weight = class_weight #Didn't use the class weights calculated above, it impacts model performance.
, validation_data=(X_validation, y_validation)
, callbacks=[best_cb])
#history.history
acc = history.history['accuracy']
val_acc = history.history['val_accuracy']
loss = history.history['loss']
val_loss = history.history['val_loss']
epochs = range(1, len(acc) + 1)
plt.figure()
plt.plot(epochs, acc, 'bo', label='Training acc')
plt.plot(epochs, val_acc, 'b', label='Validation acc')
plt.title('Training and validation accuracy')
plt.legend()
plt.figure()
plt.plot(epochs, loss, 'bo', label='Training loss')
plt.plot(epochs, val_loss, 'b', label='Validation loss')
plt.title('Training and validation loss')
plt.legend()
plt.show()
Observation:¶Both validation loss and validation accuracy stabilize around 25th epoch.
Deliverable – 5: ¶# Following code gives the best model that will be used for testing
model_b = models.load_model('model_best.h5')
# Prediction
y_pred = model_b.predict(X_test_t)
print(y_pred[1])
Observation:¶9.5842135e-01 = 0.95842135. This explains that the image belongs to class 9.
# The above code gives the probability of an image to be classified as a class.
# Now Convert those y-prediction probabilities from encdoed to a numeric class using argmax
from sklearn.metrics import accuracy_score
rounded_labels=np.argmax(y_pred, axis=1)
rounded_labels[1]
from sklearn.metrics import classification_report as cr
from sklearn.metrics import confusion_matrix as cm
print('Classification Report')
print(cr(y_test_t, rounded_labels))
# Print Label to class number mapping dataframe for easier understanding of the above and below matrices.
df_y
#Y_pred = model.predict_generator(X_test, 128 // batch_size+1)
#y_pred = np.argmax(Y_pred, axis=1)
print('Confusion Matrix')
matrix = cm(y_test_t, rounded_labels)
print(matrix)
df_cm = pd.DataFrame(matrix, index = [i for i in range(12)],
columns = [i for i in range(12)])
plt.figure(figsize = (10,7))
sns.heatmap(df_cm, annot=True, fmt='d')
Observations:¶Overall, the model performs well to label most of the images, except for "Black-grass". In the next iteration of the model, some more images from this class can be tested for model improvement.
Note: The numbers in confusion matrix may change after each run of the model.
Deliverable – 6: ¶Approach:¶Visualize predictions for x_test[2], x_test[3], x_test[33], x_test[36], x_test[59].
print("Following is the label from Y-Test")
df_y[df_y['Class_number'] == y_test_t[2]]
print("Following is the label from Y-Predict")
df_y[df_y['Class_number'] == rounded_labels[2]]
print("Following is the corresponding image")
plt.imshow(X_test_t[2])
print("Following is the label from Y-Test")
df_y[df_y['Class_number'] == y_test_t[3]]
print("Following is the label from Y-Predict")
df_y[df_y['Class_number'] == rounded_labels[3]]
print("Following is the corresponding image")
plt.imshow(X_test_t[3])
print("Following is the label from Y-Test")
df_y[df_y['Class_number'] == y_test_t[33]]
print("Following is the label from Y-Predict")
df_y[df_y['Class_number'] == rounded_labels[33]]
print("Following is the corresponding image")
plt.imshow(X_test_t[33])
print("Following is the label from Y-Test")
df_y[df_y['Class_number'] == y_test_t[36]]
print("Following is the label from Y-Predict")
df_y[df_y['Class_number'] == rounded_labels[36]]
print("Following is the corresponding image")
plt.imshow(X_test_t[36])
print("Following is the label from Y-Test")
df_y[df_y['Class_number'] == y_test_t[59]]
print("Following is the label from Y-Predict")
df_y[df_y['Class_number'] == rounded_labels[59]]
print("Following is the corresponding image")
plt.imshow(X_test_t[59])
From the samples in scope of this deliverable, one was misclassified, while others were properly labelled. The results may vary after each run of the model.